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ABSTRACT 

The second of two related pamphlets, this guide 
describes how to design an outcome study and outlines procedures for 
collecting outcome dat*. After an introduction that delineates the 
purpose of the two-part series, the pamphlet is divided into four 
major sections. The first section explains how to select a 
cost-outcome analysis. The next three sections addrfess the 
measurement of utility, benefit, and effectiveness Outcome data, 
respectively. Each self-contained section describes/ (1) the outcome 
measure, (2) when to use the analysis, (3) strengths and limitations 
of the analysis, (4) an example of the analysis, (i) an assessment of 
the reliability and validity of the outcome data, (6) how to 
calculate cost-outcome ratios, and (7) how to interpret the 
cost-outcome ratios. A list of references is included. (TE) 
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PURPOSE OP THIS COST-OUTCOME GUIDE SERIES 



.This guide series was developed in response to evaluators 1 
reported need for information on how to conduct cost-outcome 
analyses. A recent study oa the use of cost-outcome analyses by 
state education agency (SEA) evaluators {Smith, N. L, & Smith, J, 
K,, 1984) and local education agency (LEA) evaluators {Smith, a. 
K. 1984} showed 

1. in the next five years, 60 percent of all SSAs 
expect to be asked to conduct cost-outcome 
analyses, 

2. 71 percent of metropolitan LEAs expect to be 
asked to conduct cost-outcome analyses, and 

3. one of the primary impediments to the conduct of 
cost-outcome analyses is tne aosence of useful 
guides and resources. 



To assist these and other evaluators in conducting cost- 
outcome analyses, two w how-to n guides were developed. Although 
several texts currently exist on cost-outcome anaysis, they tend 
to be written in technical language or do not give sufficient 
attention to collection of outcomes as well as costs. This guide 
series supplements the existing texts by providing concise, 
readable explanations on how to conceptualize and conduct cost- 
outcome studies for program evaluation. 



The first guide introduced four types of cost-outcome 
analysis, directed the collection of cost data, and explained how 
to select the most appropriate cost-outcome analysis to answer an 
evaluation question. 



This second guide describes how to design an outcome study 
and outlines the procedures for collecting outcome data. In this 
guide, specific attention is paid to the collection of utility, 
benefit, and effectiveness data, and to the interpretation of 
study results. 



When Should This Cost-Outcome Guide 
Series Be Used ? 

Cost-outcome analyses can be applied in a variety of contexts 
to answer many different evaluation questions. The procedures 
are appropriate for evaluating mental health, healtn, and 
education programs located in community mental health centers, 
hospitals, schools, businesses, or any type of service program. 



Because of the wide *e of possible applications, it was 

necessary to focus these guides on certain applications. 
Specifically, this guide was written to direct those evaluatioi 
studies that 



• are conducted at the local or state level, 

• use experimental or quasi -experimental methods, 

• are conducted to compare two or more programs, 

• measure program outcomes. 



The guides are intended to assist the state or local level 
evaluator conduct a cost-outcome analysis for the purpose of 
programmatic improvement. For example, those interested in 
conducting a large-scale national study should consult the 
reference list of this guid* for more technical information (see 
Levin, 1983; Sugden & Williams, 1978; Thompson, i980). 



Introduction to This Gu ide 

This guide contains tour majoj sections. The first section 
explains how to select a cost-outcome analysis. The next three 
sections address the measurement of utility, benefit, and 
effectiveness outcome data.; The sections are self-contained, and 
cat: be read separately from the other sections. Each describes 

• the outcome measure, 

i 

• when to use the arjalysis, 

• strengths and limitations of the Analysis, 

• an example of the analysis, 

• an assessment of the reliability and validity >i 
the outcome data, 

• how to calculate cost-outcome ratios and, 

• how to interpret the cost-outcome ratios. 

After reading the section on selecting an outcome mea&ure, 
turn to the section pertaining to the outcome measure that best 
answers your evaluation question. 



SELECTING THE APPROPRIATE 
COST-OUTCOME ANALYSIS 



The f irst steps of a cost-outcome r*tudy are identical to the 
first steps taken in any evaluation study* For example, the 
evaluation problem must be clarified with program administrators 
before any design can be suggested. Before a cost-outcome 
analysis is selected, the evaluator must 

• clarify the evaluation question(s), 

m identify program goals and objectives relative to 
the evaluation question(s}< 

% select outcome variables to be measured, and 

# determine whether the measures can be obtained* 

These steps are a part of any evaluation, and therefore are 
undoubtedly second nature to many readers. 

To clarify the evaluation question(s) , the purpose of the 
study and the intended use of the results must be ascertained* 
This information can be obtained through informal conversations 
with the decision makers* The next step, identifying the program 
goals and objectives, is actually one of the most difficult and 
important parts of an evaluation (Rutman, 1980)* Without 
consensual agreement arac decision makers and program staff as to 
the expected program got md objectives, the evaluation would be 
directionless, and the results would probably rot be utilized* 
Patton (1978) suggests using interviews, questionnaires, and even 
decision analysis to elicit and prioritize program goals and 
objectives. 

Once the goals have been specified, it is up to the evaluator 
to suggest methods to best measure the objectives* Consider this 
simplistic example: 

Question: Which computer programming course should we 
retain— the lab class or the lecture class? 

GOAL FOR LAE CLASS : to teach computer programming 

GOAL FOR LECTURE CLASS: to teach computer programming 

PRIMARY OBJECTIVE FOR BOTH CLASSES: can write three 
utility programs 



MEASURE OPTION U written classroom test 



MEASURE OPTION 2? course completion rate 
MEASURE OPTION J: teacher rating 
MEASURE OPTION 4: student ratinos 

Now the evaluator must select the measure that is most 
reliable, valid, and available* For instance, although option 5 
may be most valid, it may be -tfee least available measure. 

How Do I Select a Cost- 
Outcome Analysi s Method ? 

The selection of a cost-outcome analysis is contingent upon 
the purpose of the study and the outcome measure* The difference 
between the three cost-outcome methods is the outcome measure used 
in each analysis. In brief, cost utility analysis is based on 
outcome measures which are estimated {e.g. , estimate of the 
probability of achieving a certain level of self-esteem). 
Cost-benefit analysis is based on outcome measures which are 
valued monetarily (e.g., income). Cost-effectiveness analysis is 
based on outcomes which are measured in effectiveness units {e.g., 
te^t scores). 

Given an outcome measure, it is simple to fina the appropriate 
analysis* Figure 1 shows an example of the steps undertaken in 
selecting a cost-outcome analysis for compacting two hypothetical 
computer programming courses in a prison. The two computer 
programming courses share the same j«>dls and objectives, so only 
one flowchart is provided. 



To select the cost-outcome analysis most appropriate for a 
given study, match to the analysis type the outcome measure which 
best answers the evaluation question. if you are evaluating two 
programs with similar outcomes, this step is simple. For example, 
if the question was "Which course is better at reducing recidivism 
in prisoners?* a cost-effectiveness analysis would be most 
appropriate. If the question was "Which course is better at 
increasing the standard of living of prisoners once they are 
released? 11 then a cost-benefit analysis would be most appropriate. 

This logic also applies to the evaluation of two different 
programs which share a common outCQme. For instance, suppose a 
counseling program and computer training program are both in 
effect in a prison and the officials want to know which program is 
better at reducing recidivism. Although the programs differ, an 



ef recti veness measure tor both could be a count of recidivism for 
participants* Here a cost-effectiveness analysis could be used, 
or a cost-benefit analysis, where benefit could be savings 
accruing from non- recidivism* If you are evaluating programs with 
different goals, an additional step is necessary to identify the 
most appropriate cost-outcome analysis* 

♦ 

Figure 1 

Selecting a Cost-Outcome Analysis to Compare 
Two Computer Program;* Which Share the 
Same Goals and Objectives 
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What If I Am Evaluating Programs 
With Different Outcomes ? 

The logic shown in the flowchart in Figure 1 should 
followed for each program being evaluated, since programs with 
different outcomes may be better suited for different types of 
cost-outcome analysis* Fortunately, selection of a cost-outcome 
analysis given different program outcomes is not problematic, 
since cost-benefit and cost-utility methods allow comparisons 
between programs with different outcomes* (Because these two 
methods convert outcomes to a common unit, that is, financial 
benefits or utility estimates, diffeient outcome measures arc made 
comparable.) Nagel (1983) provides an interesting analysis of 
uses of cost-benefit and cost-utility methods. Only 
cost-effectiveness analysis requires that outcome measures be 
alike for all programs being compared. 
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In general, the&& ace two situations in Which the ev;? tuator 
must select a util/ty or u benefit analysis because program 
outcomes are different. These two situations are shown in rows 
one and two of Figure 2* The first row in Figure 2 show* a 
situation where tne outcome tor one program i s» vaLued wouetari Ly 
ond the outcome for another is valued in utility estimates. In 
this case/ the ©valuator can conuu:t either a benefit analysis or 
a utility analysis (row one of ri^uio 2). For example, suppose 
the outcome for program A is ideally a monetary me. ,;ure (e.g., 
income from a vocational educations program) ana the outcome Lor 
program 8 is ideally a utility mtvt;vrv (e.g., improved health as a 
result of a dental prevention prout ,:m, 



One way to resolve this diierowji is that income can'ne 
translated into a utility measure o f the value uf 'increasing 
tne standard of living it) percent] .r.ij oompareu to the utility 
value of improved health* Or another way is to translate improved 
health into a benefit mea.su r« (i^y*, ji<,ount oi money saved on 
dental care) and compared to the inoirnse in income resulting fron- 
the vocational program* Which ouu *.oe is uonwr t«ni depend;, upon 
the evaluation question and the an*: ; yi; : s wn ich uouid :n< most 
meaningful to the decision ;;«u*er, 

; 

The second row in figure 2 ^rn w„ a situation wh»;rv the outcome 
for one program is valued monetarily urd the outcome foi another 
is valued in terms of effectiveness. Kith*?r tne e t i rot i veness 
measure can be converted into a monetary oenefxt, n possinle, or 
ooth the monetary benefit and t\«:r t **~ness outcomes esn he 
converted into utility values. u.-s Oonversj^u ujaici be 
accomplished by estimating tne utility uf the benefit measure and 
comparing it to the estimated utility *>t the **t t vet \ veness measure. 

Hows three and tour of Figure i demonst t ute the situations 
-••here utility analysis is the onl/ posrnoi>> unaiysx., to answer th«* 
evaluation question when outcomes tor two programs are di * lorent . 
in row three, effectiveness data js only available tor S'rogt a* A* 
rhat is, while the outcomes in* program ft could ot» H^a.sureO m 
terms of effectiveness, the data are not avdi J<i»>lt>. lp*,t«*ud t tin* 
outcomes in program B are estimated m utility* Ther^r^e, to 
make outcomes comparable, program A outcomes must he <>scirtiated as 
ut ilities also* 



In row four, the effectiveness im*x* an4*** ior two programs are 
entirely different but comparisons are requested uy the decision 
maker (e*g* f compare the et f eetivems ; d ? trading program , » : 3 <s 
math program). In this situation, outcome^ must b«* converted iut 
subjective utility values to be rp.ade comparable. Fot example, 
math and reading scores are not comparable, \ out the value of 
increasing math scores 10 percent can be compared to the value of 
increasing reading scores 10 percent, 



Vlgure 2 

Selecting a Cost Outcome Analysis for 
Programs with different Outcomes 



Outcome 



Analysis 



Program A 
monetary 
monetary 
effectiveness 



Program u 
utility 
effectiveness 
utility 



effectiveness effectiveness 
(e*g*, reading) {e.g*, math) 



benefit or utility 
benefit or utility 
utility 
utility 



Determining Whether Reliable and Valid 
Outcome Measures Can Be Obtained 

The design of a reliable and valid outcome evaluation is 
extremely complex # and many books nave been written to improve the 
conduct of evaluation studies*. Obviously, the intent of this 
guide is not to compete with such thorough works on the conduct ot 
evaluation research* The reader is encouraged to consult any ol 
these books for guidance in the design of an evaluation* The 
purpose of this guide is to assist the evaluator in obtaining 
meaningful outcome measures to use in a cost-outcome analysis* 

The next step is to design ami conduct the evaluation* The 
/' remainder of this guide describes the ptobieins of designing and 

conducting a cost^utility f cost-benef it, and cost-effectiveness 
analysis* 



COST-UTILITY ANALYSIS 



9 

ERLC 



Cost-utility (CO) analysis compares the costs of programs to 
outcomes which have been estimated. Utility, or the subjective 
value of an outcome or outcomes, can be generated by experts, 
program sponsors, or program participants for any type of outcome 
that is meaningful to the decision maker. The utility measure can 



*See Berk, 1981; Conner, 1981? Cook & Campbell, 1979; Datta 
6 Perlotf, 1979; House, 1980; Kaiton, 1983; Patton, 1978, 1981, 
1982; Rutman, 1977, 1980; Schulberg & Jerrell, 1979; Smith, 1981, 
1981, 1982; and Weiss, J 972, 1975. 
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then be combined with cost data to form a cost-utility ratio* A 
rank ordering of cost-utility ratios for two or more programs will 
indicate which program has more utility relative to its costs* 



Description of Utility Measures 
Utility measures 

• can be generated for any type of program outcome,, 

• can assess unidimensional or multidimensional 
outcomes for a given program, and 

• can be used to compare programs with different 
outcomes * 

Common measures might include the perceived utility of 

• reading at the 8th grade level, 

• completing a program, 

• increasing self-esteem, or 

• being placed in a job* 

Measures of utility can be unidimensional or multidimensional* 
An example of a unidimensional measure is parents 1 rank order ot 
their preferences for a group of different school-sponsored 
programs (Levin, 1983)* These rankings are averaged and combined 
with cost data to form cost-utility ratios which, when ranked* 
show which programs have the most cost-utility and which programs 
have the least* These comparisons of programs with different 
goals are possible because they were all valued in tne common unit 
of utility* 



An example of a multidimensional measure is the rating ot two 
alternative reading programs on four dimensions: increasing 
reading speed, increasing reading comprehension, increasing word 
knowledge, and increasing student satisfaction with reading. For 
each dimension, respondents rate their perceived utility on u 
scale of 0 (least important) to 10 (most important)* These 
ratings are averaged, and the cost utility of the two alternatives 
are compared, using composite utility measures. 



wh om, to Use Cost-Utility Analysi s 

Cost utility is a valuable planning and decision- 
making tool, particularly when 

» outcome data are not available, 

# time or money limitations restrict collection of 
outcome data, 



♦ immediate assessment of outcome is needed to 
facilitate discussion, 

♦ it is desirable to take into account the opinion 
of experts or program participants* 

The Limitations of Cost Utility Analysis 

While cost utility is viewed as useful analysis technique, 
proponents of the techniques are quick to point out the problems 
associated with using subjective estimates rather than measured 
outcome data for program evaluation and decision making* For 
instance, because utilities are generated by those invested in a 
particular program, the results would be meaningless to other 
programs* 

There have also been questions about the meaning fulness of the 
utility measures for internal decision making* Utility is a 
measure of human values; consequently, estimates of program 
outcomes are more likely to reflect personal biases than actual 
program functioning* Such subjective ratings have been shown to 
be unreliable and invalid* Following are examples* 

♦ Persona3 preferences are often "intransitive 11 
(Tversky, 1969) in that preference for A *er B 
and B over C does not insure that A is preferred 
over C* Consequently, it is often not possible 
to determine individual preference bo^ed on 
comparative ratings* 

+ People differ in the way that they scale their 

preferences* For example, on a scale of 1 to 7, 
with 7 being most preferred, Joe may rate his 
most preferred programs as 6 and 7. Sally may 
rate her most preferred programs as 4 and *>* 
Obviously, a comparison ot such ratings could b<- 
misleading* 

♦ Finally , person*! biases, conscious distortions, 
or caprice may affect the validity oi Uh> luting:;. 



Collecting the Utility Data 

Texts on cost^outcome analysis have been negligent in 
providing a detailed description of how one actually <joes about 
collecting utility data* The best available source for such 
direction can actually be found in the literature on decision 
analysis* While decision analysis is a widely recognized method 
of deriving utility measures for decision making, cost analysts 
have been slow to take advantage of the methods for use in 
cost-utility analysis* The utility methods, developed by decision 

ii. 
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Analysts, have been specifically designed to overcome many of the 
proolems with reliability and validity associated with subjective 
ratings mentioned previously. Indeed , use of decision analysis 
methods can generate utility measures which are meaningful and 
usetul (Edwards, Guttentag i, Snapper, 1975? Pitz k McKillip, 1984). 



Using Decision Analytic Techniqu es 
to Generate Utility Measur es 

Two decision analysis methods have been suggested as 
appropriate for evaluation of outcomes. These are MAUT. 
mulUattribute utility theory, (Edwards, Guttentag & Snapper. 
1975} and SMART, simple muitiattr ibute rating technique (Edwards, 
1977). The techniques used in these two methods are very similar, 
and both were developed specifically to accommodate the ne*.«ds oi 
busy decision makers working under short timelines. The steps 
involved in carrying out these methods are listed below. Detail 
about each step is provided in the example in the next sect . i of 
this guide. 



Checklist of Steps in tne Decisi on 
Analytic Generation of Utilities 

Step 1: Identity the Decision Maker, 

Step 2; Identify the Decision Question 

Step 3: Identity the Program Goals 

Step 4: Identity Relevant Program outcomes to be Estimated 

Step 5: Por Kaen Program, Rank Order trie Outcomes in Order or 
Importance 

Step 6: Rate the Outcomes to Obtain Importance Weights 

Step 7: Normalize the Importance Weights: Divide Kaeh by the 
Total Sum of Importance Weights 

Step 8: Estimate the Probability of the Occurrence of Each Outcome 

.Step 9: Calculate the Utility— Impor tance X Probability ror 
Each Goal 

Step 10: Make a Decision R commendation 



J >' 



An Example of a Cost-Utility Analysis 



Bocause utility measures are subjective and program specific, 
cost-utility analyses are not published, and therefore no known 
examples are available* There are a few examples cf MAUT analyses 
<e*g., see Edwards, et. al, 1975; Howards, 1977; Pits and 
McKillip, 1984) ♦ There are, however, no simple examples of; MAUT 
for program evaluation purposes; consequently, the following 
hypothetical example was developed* 



Because of unexpected, but not surprising, financial cutbacks, 
the County School Board must decide Whether to discontinue the 
Driver 4 s Education Program or the Dance Theater Program* Driver 's 
Education has been offered for 10 years, and is required by the 
state in order to receive a driver 's license at age 16 rather than 
age 18* The Dance Theater Program has only been in effect for 
three years, and has been extremely popular among the school *s 
minority students* For the sake of simplicity, assume that both 
programs provide service for 100 students a cerm* 



The first chree steps of the analysis were straightforward. 
The remaining steps, however, required the aid of a group ot 
students, parents, and faculty* All of the actions resulting from 
completion of steps four to nine are depicted in Table 1. 

Step 1: Identify the decision makers* 

The decision makers are the school board* 

Step 2: Identify the decision question* 

Which program should we discontinue? 

Step 3: Identify the program goals* 

The program goals were obviously different* 
Driver's Education attempts to teach driving 
skills, while Danco attempts to teach dance skills* 

Step 4: Identify relevant program goals* 

Students, parents, and faculty were called in to 
help identify which goals of the programs snould 
d be considered. Several unexpected goals were 

identified for each program, as shown in column 1 
of Table 1% 

Step 5: Rank the goals for each program* 

Again, students, parents, and faculty helped to 
order the goals by importance* Group consensus 
methods were used in arriving at the tinal order, 
although individual rankings could nave been 
derived and averaged. The rank order of outcomes 
ia shown in column A of Table I* 
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Table 1 

Steps 4 to 9 of a Utility Analysis 



Program 



Driver *s 



Dance 
Theater 



Step 4 
Identify Goals 



Step 5 
Rank Goals 



Reduce traffic fatalities Pass state exam 



Education Reduc * drinking & driving Reduce drinking & driving 
Pass state driver's exam Reduce fatalities 



Increase self-esteem 
Reduce vandalism 
Teach dance 



Teach dancing 
Reduce vandalism 
Increase self-esteem 



Step 6 Step 7 step 8 Stop * 

Rate Calculate Judge Utility 

Goals Importance Proba- (Importance 

Weight bility X Probability) 



10 

50 
90 
ISO 

10 
20 
50 
80 



10/1 SO * .06 
S0/1S0 * .33 
<>0/15G * .60 



10/80 = .12 
20/80 « .25 
50/80 * .63 



.90 
♦ 40 

.70 



.40 

.20 
,50 



.05 

,42 
*6U 

• 05 
,05 
.31 
»41 
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Step 6: Rate the goals according to importance* 

To achieve the closest possible approximation to 
ratio level ratings, raters (again, students, 
parents, and faculty) were asked to assign their 
least preferred outcome a value of 10* Each 
subsequent rating was then to be judged in terms 
of "How many more times important is this outcome 
than the least preferred outcome?" That is, 
{refer to Table 1) if passing the state driving 
exam is rated a 10 . * . how many more times 
important is the reduction of drunken driving? In 
this case, after group discussion, it was decided 
that reduction ot drunken driving was five times 
more important than passing the state driving exam, 

Ste 7: Sum the importance weights and divide each by the 
total importance sum. 

This step is shown in the fourth column in Table 1. 

Step 8: Estimate the probability of the occurence of each 
outcome* 

These probabilities were again group generated, 
although the faculty involved in each program 
provided most of the information about historical 
program effectiveness* Probabilities were 
generated as percent chance of being 
accomplished. For example, the group 'agreed that 
while most students eventually pass the state <>xam 
(90 percent chance of success), far fewer actually 
refrain from drinking and driving after taking the 
program (40 percent chance of reducing drinking 
and driving). See the fifth column tor 
probabilities generated in this exercise. 

Step 9: Calculate the utility* 

The utility of each outcome was calculated by 
multiplying the importance weights by the 
probabilities as shown in the last column of 
Table 1. These utilities were then summed to torn* 
a composite utility for each program. 

Step 10: Make the decision. 

In utility analysis, the larger the utility, the 
greater the perceived value of the program, in 
our example, the Driver's Education Program had a 
utility of .60, as compared to a utility of .41 
for the Dance Theater Program. Theae figures, 
however, should not be interpreted for decision 
making purposes until the reliability and validity 
of the data have been considered. The next two 
questions deal with fches* important issue;. 
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{ft>w MAUT Improves the Integrity 
_ Qf_ the Utility Data 

Recall the three criticisms of utility analysis having to do 
with rating credibility: intransitivi ty , scaling differences, and 
rater caprice. Systematic application of MAUT helps to, minimize 
thtse problems. The use of group input should minimize the chance 
of intransitivities, since group interaction is more likely to 
catch such an inconsistency than would an individual rater. The 
calculation of the weighted importance values eliminates the 
problem of scaling differences. Finally, group interaction would 
also help insure minimal caprice in the ratings, in fact, a group 
decision or individual ratings averaged, would help insure 
Stability of ratings. 



Assessing the Reliability and 
Validity of the Utility Data 

The validity of the data is contingent upon the generation of 
meaningful program outcomes, while the reliability concerns 
consistency of the numerical data associated with those outcomes. 
Levin (1983) suggests that, to ennance validity, it is important 
/ that the utility ratings be generated by those who are most 

- / familiar with the program, and who will be affected by the 

changes, in our example, obtaining information from students, 
7 parents, and faculty would help to insure that the utilities are a 

I valid reflection ot the values of individuals involved in the 

program. Similarly, using group processes (or averaging 
individual data) would increase the chance that the ratings are 
stable (reliable). Individual ratings are much more likely to be 
unrealizable than are group ratings. 



Calculating the Cost-Utility Ratios 

Cost-utility ratios are calculated by combining the cost data 
with the utility data. Suppose, in this example, the total 
program cost was calculated at $2,000 for Driver's Education and 
$1,000 for Dance Theater. As calculated in Table 1, the utility 
for Driver's Education is .60 and the utility for Dance Theater is 
.41. (In order co obtain more intuitively meaningful figures, 
these utilities were multiplied by 100 for this example.) The 
cost utility of Driver's Education is 2000/60 * 33.33, and for 
Dance Theater is 1000/41 » 24.39. These figures are shown in 
Table 2. Notice these are program level data* total cost versus 
total utility. Per-participant a level data can also be generated 
by dividing both total figures by the number of participants It 
is important not to mix total figures with per-participant 
figures. For utility analysis, however, program level data is 
often most meaningful. 
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Table 2 

Calculating the Cost-Utility Ratio 

Composite 
Total Program Utility 
Program Cost Measures X 100* CU Rat io 

Driver's Education $2,000 .60 X 100 * 60 33*33 

n « 100 

Dance Theater $1,000 .41 X 100 * 41 24.39 

n » 100 



♦Multiplied by 100 to obtain whole numbers. 



Interpreting the Cost-Utility Ratios 

The cost-utility (CU) ratio is useful only when compared to 
another CU ratio. There is no intuitive interpretation ot a 
singular ratio. Recall that a higher utility is preferred when 
just examining utility values. However, when comparing two 
ratios , the lower ratio reflects less cost per unit of utility. 
In this case, while the utilities for Driver* s Education is higher 
than for Dance Theater, the CU analysis points to the retention of 
the Dance Theater rather than the Driver *s Education Program. 



COST-BENEFIT ANALYSIS 



Cos1;-benef it (CB) analysis compares the costs of programs to 
outcomes which have been valued monetarily. This method has often 
been used in the assessment of nationwide health policy, and in 
environmental policy studies (Thompson*. 1980). It has had fewer 
applications in program evaluation. One reason that cost-benefit 
analysis |has not been widely used in program evaluation is that it 
is difficult to assign monetary values to social service program 
outcomes that are not traded in the market (Thompson, 1980). For 
example, it is difficult to monetarily value an increase in 
reading comprehension. 

Description of Benefit Measures 

Thompson (1980) has identified five categories of methods for 
valuing program outcomes monetarily. These categories are as 
follows. 




Money measurement. 



This is a simple assessment of the amount of money saved or 

earned as a result of participating in a program/ It is the 

simplest and most straightforward valuing method. Examples of 
this type of measure include: 

Evaluation, Question 

How much does participation in 
vocational education increase the 
student f s income? 

How much does the health serening 
program save participants in terms 
of Income lost due to illness? 

How much money is saved by taking 
a bus versus a commuter train? 

Market valuing. 

This is very similar to money measurement, except that it 
involves the transformation of an outcome into a monetary 
value based on market rates. Per example, the cost of a new 
school might be compared to the benefit of increasing land 
value on the adjacent real estate* 

Econometric estimations* 

This valuing method requires an under standing of the supply 
and demand curve, and is not easily calculated by evaluators. 
econometric estimations are most often used to figure the 
value of a human life for federal level policy decisions 
{Rhoades, 1978). In brief, there are several ways to estimate 
how much a life is worth, such as 

how much an individual will pay to 
live 

how much an acquaintance would pay 
to enable a person to live 

the maximum thut could be paid to 
keep someone a Live without 
negatively affecting someone else 

life value depends upon how much 

money u» .spent 

how much an individual would earn 
in a lifetime 



Outcome Measure 
income 

income saved 
money saved 



# own life value 

• acquaintanceship lite 
value 

* pareto life value 

# consumptive- life value 

• earnings life value 



# willingness to pay rational average someone will pay 

for chance to live 

+ social life value amount a group of people think the 

government should pay to keep 
someone in treatment 

4. Hypothetical questions. 

This method asks an individual or group a hypothetical 
question, such as "How willing would you be to pay for this 
service * * * or to avoid this tragedy . * * or to attend a 
session on this versus that topic?* For example , a 
willingness-to-pay question is, "How much would you be willing 
to pay to attend a session on assert iveness training versus a 
session on hypnosis?* 

5. Observing political choices* 

This method uses social and legislative action as a gauge for 
the financial value* For example, if the equal rights 
amendment passes, then the same methods for valuing a life 
based on life earnings can be used for men and women. Until 
then, however, different methods must be used to calculate 
earnings for men and women* 



There are significant problems with all of these methods, and 
their use is generally not recommended {Thompson, 1980). Except 
for money measurement and market valuing, most methods are 
problematic in that they contain systematic biases, 
underestimations, and distortions* Rothenberg (1975) commented 
eloquently on the Statf of data for cost-benefit analysis: 

/ / 

* * . regardless of its (cost-benef it analysis) 
methodological claims, its practical usefulness will 
be most decisively at the mercy of the availability of 
data* Very serious inadequacy of relevant data exists 
in almost every area for which cost-benefit analyses 
have been undertaken* (p* 88) 



When to Use Cost-Benefit Analysis 

There are times when program outcomes are verbalized in terms 
of monetary value, such as in the evaluation of a vocational 
education program, to see its effect on the incomes of 
participants* Cost-benefit analysis is most useful for evaluating 
programs with outcomes which can logically be valued monetarily, 
or when the decision makers specifically request financial outcome 
data* In addition, because outcomes are valued in a common unit 
(monetary value), programs with different outcomes are 
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compatible. So, like cost-utility analysis, a mechanic's program 
could be compared to a program Cor unwed high school mothers if 
the monetary value of each program can be meaningfully determined. 

Example of Cost-Benefit Analysis 

There are many published examples of cost-benefit analysis 
conducted f- v purpose of program decision making. Most 
examples are of a policy nature, and assess benefit on a societal 
rather than programmatic level. Andrieu (197?) provides an 
interesting example of a N cost-benefit analysis of manpower 
training programs, and uses yearly earnings as a benefit measure. 
Malitz (1984) recently conducted a cost-benefit analysis of family 
planning services in Texas. Maiitz used econometric estimates of 
the costs of the number of pregnancies, births, abortions, and 
miscarriages averted. 



As an example of cost-benefit analysis for programmatic 
decision making, consider this hypothetical decision problem about 
Driver's Education versus tho Dance Theater. 



Because of unexpected, but not surprising, financial cutbacks, 
the County School Board must decide whether to discontinue the 
Driver's Education Program, or the Dance Theater Program. 
Driver's Education has been offered \or. 10 years, and is required 
by the state in order to receive a driver's license at age 16 
rather than age 18. The Dance Theater Program has only been in 
effect for three years, and has been extremely popular among the 
school's minority students. For the sake of simplicity, assume 
that both programs provide service for 100 students a term. 

The selection of a benefit measure should be based on a 
measure that would be meaningful to decision makers. The decision 
makers have asked which program generates the most revenue for the 
school. The benefit generated by the Dance Theater can be easily 
measured by tabulating door charges from shows. 



The benefit of the Driver's Education program is not as easily 
measured. The decision makers solved the measurement problem by 
wondering if students who could drive would be more likely to 
attend school events. Since, without driver education, a student 
cannot take the state licensing exam until the age of 18, it is 
theoretically possible that by taking driver's education, the 
student may gain two years of opportunity to attend school 
events. Benefit for this evaluation could be defined as fees 
collected from students attending events as a result of having a 
driver's license. Students who took driver's education last year 
were surveyed and asked a hypothetical question about their 
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attendance now versus their estimated attendance if they did not 
have their license* increases in school revenue were projected, 
based on these responses, and fees collected at the school 
events* The benefit derived from the Driver 's Education program 
was calculated at $600 and ior the Dance Theater at $1000, 



Evaluating the Reliability and 
Validity- of the Benefit Measures 

One of the first issues to consider about the validity of 
benefit measures is whether they accurately reflect program 
functioning* Obviously, the direct market value of fees collected 
at Dance Theater shows is not likely to be questioned. The 
estimation of events attended as a result of having a driver's 
license is much more suspect. Prior to the collection of this 
particular measure, the evaluator should have confirmed with the 
decision makers that this measure would be meaningful to them. If 
it is meaningful, then the evaluator should examine the method 
used for sampling students, and consider the possibility oi 
systematic biases which may affect responses* 



Calculating the Co st- Benefit Ratio 

Once the benefit and cost data have been collected, it is 
simple to combir^ the two to form a cost-benefit ratio. Table 3 
demonstrates that the total program cost for Driver f s Education 
($2,000) when divided by the total benefit ($600) results in a 
ratio of 3.33. The cost-benefit ratio for the Dance Theater is 
.33 (1000 S- 3000). The interpretation of these figures is in the 
next section. This example also shows that when student fees are 
used as a measure of benefit, the Driver's Education program 
results in a loss of $1,400 (net benefit a benefit - cost), while 
the Dance Theater earns $2,000. 



Notice that these ratios are based on program level data. 
Total program cost is being compared to total program benefit. It 
can also be useful to calculate the cost-per-participant for each 
program. To calculate per*-par ticipant figures, divide the total 
cost by the number of participants. 



Interpreting the Cost-Benefit Ratio 

The net benefits show that the costs of driver's education far 
exceed the benefits— when benefits are based on money generated 
from students 9 fees. In contrast, the Dance Theater benefits far 
exceed the costs— when benefits are based on money generated from 
student fees. Another way to interpret Miese results is to 
translate the ratios into cost terms. Driver's Education spent 
$3% 33 to generate $1.00 of income for the school, whereas the 



Dance Theater spent only 33*r to generate $1.00 of revenue for the 
school. Clearly, if decision makers are primarily concerned with 
the revenue produced by these programs, they would retain the 
Dance theater and discontinue Driver's Education. 



Table 3 

Calculating a Total Cost-Benefit Ratio 



Program 



Total 
Program 

Cost 



Driver's Education $2,000 
n « 100 



Total 
Benefit 



CB Ratio Net Benefit 



$600 



3.33 



-1400 



Dance Theater 
n = 100 



$1,000 



$3 , 000 



+2000 



COST-EFFECTIVENESS ANALYSIS 



Cost-effectiveness analysis compares the costs of a program to 
outcomes measured in units of effectiveness, when evaluators 
think of program outcome measures, they automatically think of 
effectiveness measures. Historically, effectiveness measures such 
as test scores, performance measu-es, or other ratings, have been 
used as a barometer of how well a program is achieving its goals. 
Analysis of effectiveness data can be used formatively to improve 
program functioning, or sumroatively to make programmatic 
decisions. Outcomes measured in effectiveness terms are: 

• empirical and therefore replicable, 

• meaningful to decision makers and participants alike, 

• often suitaole for complex statistical analyses, 

• sometimes available as a result of routine program 
monitoring. 



Descriptions of Effectiveness Outcome Measures 

S 

9 Measures of effectiveness can be obtained for outcomes of 
cognitive skills and psychomotor skills, and for affective 
variables, using tests, observation, surveys, and program 
records. Examples of outcome measures include the following. 



e difference between pretest and posttest scores * 

* final test scores* 

* program completions, 

* job placement, 
e satisfaction* 

* 30b performance* 

* morbidity or mortality* 

* number of deaths averted* 

* recidivism rate, 

* number of clients served* 



Obtaining Meaningful Outcome Measures 

Any text on evaluation addresses the critical issues of 
measuring and collecting effectiveness data. The true difficulty 
in conducting a valid and reliable cost-effectiveness analysis is 
obtaining meaningful measures. When considering a test* it is 
best to use one that has been standardized and shown to be 
reliable and valid (Nunnally* 1975; Nunnally & Wilson* 1975)* 
rather than to develop your own measure. If you do develop a 
test* there are important procedures necessary to insure that the 
collected data accurately reflect prooram effectiveness* Cole and 
Nitko (1981) outline these important issues that go beyond the 
scop^ of % this guide* Similarly* for using archival dat£* 
interviews* or surveys* Weiss (1975) and Weinstein (1975) have 
helpful suggestions. Perhaps the simplest effectiveness measures 
would be tabulations of number of clients served* number of 
dropouts* or other similar indicators of program effects. 



To review* when deciding upon effectiveness measures r consider 
these questions, 

• Is the measure meaningful to decision makers? 

• Is the measure available? 
% Is the measure reliable? 

• Is the measure valid? 



Limitations of Cost-Effectiveness Analysis 

Cost-effectiveness analysis* unlike cost-utility *>nd cost- 
benefit analyses* requires that comparisons be made on the same 
outcomes. For example* it is meaningful to report that at the end 
of the term students in computer assisted instruction (CAI) 
reading got 80 percent correct on the final exam, while students 
receiving peer tutoring in reading got only 60 pc . correct on 
the same exam* Obviously* unless some other variu. *e was 
affecting student scores* CAI was more effective in increasing 
reading ability* Such a comparison* however* cannot be m^de 
between reading and health programs where students in reading get 
80 percent correct on the reading exam and students in health 




education get 50 percent correct on the health exam. Since the 
outcome measures of the two pro-ams are dissimilar, it cannot be 
concluded that one program is more effective than the other. 



Examples of a Cost-Effectiveness Evaluation 

There are many examples of CE studies that have been 
published. A few of these include; ' 



Author 



Doherty & Hicks 
(1977) 



Klarman, 
Francis & 
Rosenthal (1968) 

Quinn, 

Van Mondfrans & 
wo r then (1984) 



Evaluation Question 

Effectiveness of health- 
care programs for the 
elderly 

Cost-effectiveness of 
treatment of chronic 
venal disease 

Cost-effectiveness of two 
math programs 



Outcome Measure 

Scale of daily 
activities 



Mortality 



Achievement test 
scores 



Skellie, Mobley, 
«. Coan (1982) 

Yates, B. (1978) 



Cost-effectiveness of long- 
term health care 



Dajs ot survival 



Cost-effectiveness of weight- Pounds lost 
loss prog rant 



Reliability and Validity of 
Effectiveness Outcome Data 

If effectiveness outcome data are not reliable (stable) and 
valid (meaningful measure of program), they will be of little us 
in evaluation. Cook and Campbell (1979) provide an excellent 
discussion of reliability and validity issues in quasi- 
experimental studies. The following is a checklist of issues 
intended to be used prior to interpretation of data collected in 
an outcome study where participants are not randomly assigned to 
programs. 



Reliability. Is the outcome measure stable? If outcomes 
available for cnly a few participants, spurious effects may 
result. However, if data are aggregated for a number of 
participants, the reliability is increased. 
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Validity . Internal validity refers to the relationship 
between the program implementation -and the outcome* There are 
many variables that can complicate the interpretation of an 
outcome and bring to question whether the outcome measure 
accurately represents program effectiveness. The following 
summarizes the discussion about threats to internal validity by 
Cook and Campbell (1979)* 



Each threat is described using a hypothetical thira-graue 
reading program which uses the difference between pretest and 
posttest administration of a standardised achievement test as **n 
outcome measure* 

History: Did any other factor contribute to the observed 
change in the program participant? For example, was a 
lunch progr:am implemented at the same time that might 
have af fecte^^clvievement scores? 

Testing: Did students r>e<UJ. the pretest at the time of 
the posttest and as a result do better? 

h» Instrumentation: Is it possible that the test was too 

hard ("floor* effect), or too easy ("ceiling* effect) at 
either the pretest or posttest? 

Statistical Regression: If students were selected for 
participation in the prdgram because of low/ achievement 
scores or deprived home environments, then 4f* increase in 
test scores between the pretest and the posttest could 
occur simply as a statistical artifact* 

Selection: If comparing two similar programs, consider 
whether the children in the two programs were dissimilar 
for any reason (e»g», neighborhood differences, age 
differences, sex differences)* 

Mortality: Ask if gains might be spuriously distorted 
because low achieving or high achieving students dropped 
out of the program* 



These are just a few of the basic threats to internal 
validity* Evaluators unfamiliar with these or other threats to 
internal validity should consult Cook and Campbell (1979, 
pp. 53-55) for additional information* 



Calculation of Cost-Effectiveness Ratios 

Once the outcome data have been collected and examined for 
reliability and validity, it is a simple matter to combine the 
cost with the outcome data to form a ratio* Suppose computer 
assisted instruction and peer tutoring reading programs are being 



O 24 or 



"ERIC 



6o 



compared for cost-effectiveness, in this example it cost $500 per 
student to provide the computer assisted instruction program and 
the average student got 70 points correct on the final exam. This 
computes to a cost-effectiveness ratio of 7.1. The peer tutoring 
program was much less expensive per student ($100) and the average 
student got 50 points correct on the final exam. This computes to 
a cost-effectiveness ratio of 2. Table 5 depicts these figures 
and calculations. 



Table 5 

Calculating the Cosl-Etfectiveness Ratio 



Program 


Pre ji'am 
Cost Per 
Student - 


Effectiveness 
Cost Per 
Student 


CE Ratio 


CAI reading 


$500 


70 points 


7,1 






correct 




Peer tutoring 


$100 


50 points 


2 


reading 




correct 





Notice that these figures are represented in per-participant 
figures. Per-participant figures are determined by dividing 
program-level data (e.g., costs or outcomes) by the number of 
participants. It is important not to mix program level costs with 
participant level outcomes, since the ratio would be misleading. 
Either per-participant or program level ratios are meaningful; tne 
selection of either should be based. on the types of answers most 
useful to the decision makers. 



TnteTfpretation of Cost-Effectiveness Ratios 

A cost-effectiveness ratio can be interpreted as the cost to 
produce a unit of effectiveness. It cost about $7 per point with 
computer assisted instruction versus only $2 per point with peer 
tutoring. 



Of course, there is a trade-off here. While peer tutoring is 
less expensive per point, students in computer assisted 
instruction received an average of 20 more points at the end of 
the term. This creates an ethical dilemma for the decision 
makers: now that they have been provided with the data, they must 
use it wisely in making decisions. 
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